Assessing Test Quality

With the success of agile methodologies, Testing has become a common and important activity in the development of software projects. Large and automated test-suites ensure that the system is behaving as expected. Moreover, tests also offer a live documentation for the code and can be used to understand foreign code. However, as the system evolves, tests need to evolve as well to keep up with the system, and as the test suite grows larger, the effort invested into maintaining tests becomes a significant activity. In this context, the quality of tests becomes an important issue, as developers need to assess and understand the tests they have to maintain. While testing has grown to be popular and well supported by today’s IDEs, methodologies and tools trying to assess the quality of tests are still poorly or not at all integrated into the testing process. Most important, there has been no attempts yet to concretely measure the quality of a test by detecting design flaws of the test code, so called Test Smells. We contribute to the research of testing methodologies by measuring and assessing the quality of tests. In particular we analyze Test Smells and define a set of criteria to determine test quality. We evaluate our results in a large case-study and present TestLint, an approach to automatically detect Test Smells. We provide a bundle of tools that tightly integrate source-code development, automated testing and quality assessment of tests.